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ABSTRACT 

As the technology of the workplace changes, U.S. 
workers are called upon to have increasingly high workplace literacy 
skills in order to remain competitive. These skills encompass more 
than academic skills of reading and arithmetic; they involve complex 
applications of reasoning and problem-solving skills. In order to 
meet the need for workers with such skills, businesses often provide 
workplace literacy training programs. Such programs must be evaluated 
to see how well they are carrying out their mission and how they can 
be changed to meet the needs of employers and employees. Evaluation 
methods vary as the outcome sought varies. Companies generally 
evaluate training on four levels that measure: (1) to what extent the 
training meets organizational needs; (2) how well the employees 
master the content of the program; (3) how mastery of training 
transfers to, and has a positive impact on, job performance; and (4) 
to what extent changed job performance results in cost benefit to the 
organization. Evaluation criteria can be formulated based on these 
four levels of employer needs. Whatever type of program evaluation is 
used, it is critical that the data provide evidence that the program 
is or is not effective, based on the values of the organization. A 
true measure of program success is the willingness of the 
organization to expand it. (Contains 21 references.) (KC) 
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How Do You Know If It's Working? 

Evaluating the Effectiveness of 
Workplace Literacy Programs 

Workplace literacy programs often are perceived as a potential means for upgrading 
the skills of the labor force in order to maintain a competitive role for Americans in a global 
economy. As a wide range of replicable workplace literacy program models emerges to fit 
the various needs of diverse partnerships and conditions, employer and employee requests 
for information about such programs continue to increase. No one, however, wants to 
buy a "pig in a poke;" prior to investing time or money, employees and employers alike 
want to know what causes programs to be effective and which programs have proven to be 
most successful. 

In the workplace, literacy programs are viewed as components of "training," 
usually conducted under the direction of personnel or human resource development 
departments. As a function of training, workplace literacy programs need to be evaluated 
differently from other adult education programs. This is because training, by definition, is 
provided in response to organizational needs as well as those of the individual; education is 
a response to individual needs alone. Any effective training program, e.g., a workplace 
literacy program, demonstrates positive impact both on organizational needs and on those 
of individual participants. Workplace literacy programs stem from the desire for joint 
survival by both employees and the organizations for which they work (Fields, 1986). 
Even those workplace literacy programs now sponsored by labor organizations have 
expanded beyond adult basic education courses and tuition reimbursement plans as a direct 
result of member requests for augmentation of company training in new job requirements 
(Philippi, 1992). In order to understand why literacy in the workplace is considered 
"training," rather than "education," it is important to examine the underlying reasons for the 
existence of this new field. 

Whv Literacy Pmp* m fa j n thft WnrWp1 arff 

Today's economy and workplace environment arc changing rapidly, accelerating 
the need for workers to use literacy skills as they perform job tasks. The shifts toward 
self-directed teamwork and emphases on quality production and services necessary for 
competing in a global marketplace have created job tasks that employ numerous 
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applications of literacy skills. Both large and small companies are requiring many new job 
tasks of their employees in efforts to remain in business and to leapfrog their competitors. 

Changes now taking place in the workplace center around improvements in product 
and service quality and increased worker responsibility. Most companies attempt to make 
processes more efficient and profitable through: 

• shifts in responsibility, such as downsizing, cross-training with 
accompanying certification processes, and self-directed work teams; 

• upgrades in technically sophisticated equipment, such as computer 
terminals, robotics and program logic controlled machinery; and, 

• collection and analysis of product and service quality statistics to 
measure customer satisfaction, cycle time and consistency. 

The motivation for all of these efforts is an organization's bottom line— and many 
companies are "running scared." Risking survival on a new piece of equipment or a shift 
in organizational structure causes more and more companies concern as they operate with 
leaner staff resources, greater demands from customers and tighter timelines. 
Organizations are increasingly uncertain of the capacity of their current workforce and 
prospective new hires to master the skills necessary to accomplish these critical changes. 
In many cases, organizations must rely on a workforce that includes incumbent employees 
who ranked low academically in school or who have limited proficiency in English. 

Many of the countries with which America competes in the Asian and European 
economic communities have been able to begin afresh as they rebuild new economic 
systems after obliteration caused by war. In contrast, America is trying to transition from 
outdated hierarchical work systems to quality-oriented, flattened organizations withou t a 
hiatus resulting from the economic collapse of our existing system. The enormity of this 
task, currently faced by American employers and employees, has been compared to that of 
an electrician trying to "rewire the house while the lights are still on." Because of the 
necessity to remain an aggressive player in a global marketplace during the reskilling of our 
labor force, the workplace has become a vast learning community with immediate and 
critical needs. 

i 
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To accomplish new workplace tasks effectively and to improve current job 
performance, employers have begun to realize the necessity for providing workers with 
training in job literacy skills. Most employees also feel the need for, and often ask for a 
"brush up" or special training to prepare themselves to assume control of their work 
environment. The average worker is almost 40 years old. For years he has performed the 
same job tasks, but as his job responsibilities change with upgrades in technology and 
shifts in management structure, he finds himself no longer equipped with the skills he 
needs to tackle new job tasks. He is classified as an "intermediate level literate," no longer 
able to function competently in a changing workplace environment without additional 
literacy skills training. 

Traditional educational systems in communities are designed to meet the needs of 
individuals, not organizations. For that reason, the specific needs of workers and 
organizations are being met with workplace literacy programs that enable participants to 
transfer job-linked literacy skill applications to the performance of their job tasks. 

The Relationship bet ween Training in Literacy Slrills 
and Performanre of Jofr J^fa 

For every job there are tasks that are critical to its performance. The varying 
degrees to which workers can perform these critical tasks determine their levels of job 
proficiency and, collectively, determine the quality of the organization and of the labor 
force. Competent performance of job tasks requires more than knowledge of job content 
Superior workers are those who are able to identify job needs and efficiently use basic 
skills applications, (i.e., reading, writing, speaking, listening, computation, problem 
solving), to complete job tasks (Mikulecky, 1987). These skill applications arc known as 
"workplace literacy." 

Workplace literacy tasks require different applications of skills from those taught 
and used in academic contexts. Competent workers must now be able to use: 

1.), job reading processes for locating information and for using higher level 
thinking strategies to draw conclusions from multiple sources in order to 
problem solve; 
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2. )* occupational speaking and writing processes for organizing clear 
communication of ideas and for mastering those thinking skills that enable 
clarification, analysis, elaboration and extension of spoken or written 
information; and, 

3. ), workplace applications of mathematical concepts and processes for 
calculating information, collecting data and solving problems that go beyond 
basic number concepts and computation skill drill and enable workers to 
acquire proficiency levels in reasoning and interpretation. (Philippi, 1987-88) 

These workplace skill applications all require the use of cognitive strategies and arc seldom 
used in isolation, but generally cluster in combinations related to the performance of 
specific job tasks. 

Even those workers who have mastered academic literacy skills are seldom 
prepared for the ways they will need to use those skills to competently perform the current 
and future job tasks in which these embedded skills are found in the workplace* To solve 
this problem, many employers and labor organizations have begun to offer literacy skills 
training programs to supplement technical training on new job tasks* The most successful 
of these are programs that are designed to facilitate maximum transfer from the learning 
situation to job performance. These are programs that are built on actual job scenarios and 
that utilize job materials as a vehicle to teach the metacognitive and cognitive strategies that 
competent workers use. Instructional objectives are not content-specific, but rather focus 
on the thinking processes that are embedded in real-life job task examples with which 
workers are familiar. Using this type of functionally contextual instructional material in 
workplace literacy training is important to both the organization and to individual 
participants. For the organization, the processing performed in instruction and in the tasks 
to which learning is to be transferred is very similar, similarity must be high in order to 
achieve subsequent retrieval and application of knowledge; if the processes are different, 
procedures learned in training yield no transfer, or even negative transfer (Bransford and 
Franks, 1976; Tulving and Thompson, 1973). For the individual, it ensures that the 
instruction will be meaningful in terms of prior knowledge structures; using existing 
"mental hooks," or schemata, derived from work environment and experience to attach new 
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information helps ease the process of assimilating old and new knowledge (Farr, Carey and 
Tone, 1985; Fingeret, 1984; Shoemaker, 1967; Valentine, 1985). 

The Purposes of Program Evaluation 

Evaluation is generally thou;* V to be the concluding step in a series of program 
components, the completion of a training cycle. It is often described as a summation, in 
terms of "outcovnes," and is ftequently viewed as a somewhat frivolous program after- 
thought or a small but necessary requirement fulfilled in order to remain in compliance with 
regulations of external funding sources. In reality, evaluation is an integral pan of any 
program design; it dictates and monitors data collection to determine summative 
cyclical.outcomes as they relate to an ongoing formative program process. Properly carried 
out, the process of program evaluation should provide information that improves the 
effectiveness of program design, development, implementation, and operation (Sticht, 
1991). It should also provide hard data that identifies indicators of program effectiveness, 
measures change in terms of those indicators, and works toward establishing program 
standards from the results (Condelli, 1992). 

Evaluation can serve a number of purposes. Five main categories are described by 
Bramley in Evaluating Training Effectiveness: Translatin g Theory into Prartjffi , (1991, pp. 
85- 108). Bramley suggests that evaluation is used most commonly as a means for 
providing feedback that enables quality control over the design and delivery of program 
activities. Feedback to participants is an essential part of thejearning process. Feedback to 
providers and instructors helps developers make appropriate modifications to current and 
planned future offerings. Evaluation feedback provides a means for clarifying program 
goals and objectives, observing whether or not they have been achieved, and producing 
information that can be used by program decision makers to improve a program's capacity 
to achieve its goals. 

Another purpose for evaluation is to develop policy and practice criteria, as they 
relate to a program sponsor's goals. This form of evaluation is called "control evaluation" 
and attempts to measure program worth and cost to the sponsor through comparative 
studies of different combinations of methods for attacking specified problems. Evaluation 
conducted for the purpose of control extends beyond feedback, posing comparative 
questions, and tends to result in reports containing many recommendations. 

U.S. Department of Education/Office of Vocational and Adult Education 
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On a slightly more formal level, research evaluation can be conducted to draw more 
universal conclusions from data. When evaluation is designed as research, its purpose is to 
add knowledge of principles and practice to general application in the field Research 
evaluation attempts to draw specific conclusions from a carefully controlled situation 
(internal validity) and determine the extent to which those conclusions apply to other 
situations (external vanity). 

Yet another purpose of evaluation is intervention. Involving program participants, 
sponsors and providers, i.e., the "stakeholders," in goal setting, data collection and 
debriefing can redefine the sharing of program responsibility for learning. This is a 
powerful method of intervening in an organization's procedures when change in the way 
things are done is the desired outcome. 

Finally, evaluation can be used as power. Evaluation information is potentially 
powerful in a political way within an organization or for the group sponsoring a program. 
It is next to impossible to avoid the use of an evaluation in this way; however, it does 
underscore the responsibility of any evaluator to ensure that the information being used is 
based on a thorough and careful study, rather than reports based on anecdotal evidence 
alone (Randall, 1960). 

U.S. federal and state governments fund numerous demonstration projects each 
year in attempts to identify those workplace literacy program components and models that 
provide effective solutions to this national problem. The published purpose of the national 
Workplace Literacy Program grants is "to increase the productivity of the workforce 
through the improvement of literacy skills in the workplace" (Federal Register, 1989). 
Private, public and military organizations also fund numerous workplace literacy programs 
for similar reasons. Feedback evaluation is important to providers, business and labor 
partners, and participants involved in each of these endeavors throughout the life of their 
projects. Evaluators both internal and external to these projects can gather and analyze data 
that demonstrates appropriateness of goals and outcomes, along with levels of effectiveness 
for program design, development, implementation and process. Comparing evaluation 
reports and information among projects could result in recommendations that establish 
policy and practice controls for future projects. Creating project designs that operate in 
carefully controlled situations could result in research findings that add knowledge of 
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principles and practices to general application in the field. Participation in evaluation by all 
stakeholders in projects could lead to positive intervention in existing education and training 
procedures; and, responsible evaluations that collect hard data could produce positive, 
rather than negative, conclusions about the value of workplace literacy. 

Defining Evaluation 

Much of the underlying theory that exists in the modern field of evaluation has 
evolved from the practices used in educational evaluation in the United States. In the 
1930s, strong emphasis was placed on the need for objectives by which to organize school 
curricula (Tyler, 1932; 1950). Objectives became the basis for planning and guiding 
instruction, for preparing test and assessment procedures, and for systematic program 
evaluation. This led to the comparison of programs and the development of standards by 
which all educational programs were measured across the country. 

In the 1960s, it became apparent that the development of new curricula would need 
a new type of evaluation that provided feedback to program developers during the 
development period (Cronbach, 1963; Striven, 1967). From this era came the terms 
"formative evaluation," concerned with program development, and "summative 
evaluation," concerned with program worth. This distinction allowed for program 
evaluation that focused on the performance characteristics of an individual developing 
program, rather than on comparison with others. Evaluation of workplace literacy 
programs should emphasize formative evaluation while including cyclical summative 
evaluations ( based on the premise that the workplace will continue to experience 
accelerating change and training in new skill applications will be an ongoing need). 
Evaluation of workplace literacy programs needs to measure program effectiveness 
independently, i.e., by comparing each actual program to its own stated goals, rather than 
comparing it to other programs. This is because each workplace literacy program designed 
in response to company and worker needs is unique; no two organizations have exactly the 
same set of critical job tasks to be addressed by instruction. 

By the mid 1970s, a new version of independent program evaluation had 
developed, called "responsive evaluation" in which the eyaluator is primarily concerned 
with program effects in relation to the interests and goals of the program "stakeholders" 
(Stake, 1975). Responsive evaluation begins by identifying the goals each stakeholder 
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group has for the program. This is followed by evaluator observations of program 
materials and implementation, which are compared to the various goal expressions. The 
evaluator identifies issues and concerns, collecting data to satisfy these with a variety of 
instruments. Collected information is organized into themes and issues and concerns are 
matched to audiences. "Progressive focussing" can be used to systematically reduce the 
breadth of inquiry and concentrate attention on emerging issues (Parlett and Hamilton, 
1977). Responsive evaluation has potentially strong features for evaluating workplace 
literacy training because it attempts to take into account the iijjerests of various groups of 
stakeholders, rather than just the program developers or sponsors, and collect data that 
meets their needs. 

If the end goal of workplace literacy is to improve the productivity of the 
workforce, programs should be viewed as an integral part of organizational training- a 
long-term investment in the building of a highly skilled labor force. Evaluation design, as a 
part of such programs, needs to determine program effectiveness according to the value 
yardsticks that business and industry apply to training. Quality indicators, measures and 
performance standards for individual programs need to be identified from the workplace 
community, rather than from educational environments. 

Whenever companies evaluate training, they generally assess effectiveness on four 
different levels: 

Level I- To what extent does the proposed training program match a critical 
organizational training need? 

Level II- To what extent do employees who participate in the training master the 
content of the program? 

Level IH- To what extent does mastery of training transfer and impact positively on 
job performance? 

Level IV- To what extent does changed job performance result in cost benefit to 
the organization? 
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These evaluation levels differ significantly from traditional education evaluation 
criteria. Much of evaluation of educational programs focuses on individual progress 
toward universally defined standards, i.e., grade levels or letter grades, which sometimes 
is then compared to group achievement. In addition, feedback from individual participants 
and instructors is sometimes solicited on a structured basis to determine the need for 
adjustments to materials, scheduling or instructional techniques. 

The relationship between training and evaluation is illustrated in Figure 1 below: 

Figure 1: Training/Evaluation Relationship 
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When educators enter the workplace as providers of workplace literacy programs, it 
is important to conduct evaluation that provides meaningful information based on the 
existing values of the workplace community. Level I evaluation criteria for a workplace 
literacy program can be effectively met when internal or external program providers: 
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♦ work with employers and employees to determine organizationally 
critical job tasks and program goals; 

♦ identify the skill applications needed to perform those tasks competently; 

♦ develop a program that instructs in those skill applications through the 
use of job scenarios and materials as the vehicle for instruction; and 

♦ collect pie- and postprogram data on employee expectations and 
satisfaction with program content and delivery. 

Level n evaluation criteria can be effectively met when program providers: 

♦ collect representative examples of participants 1 work to demonstrate 
progress toward program instructional goals; and 

♦ develop and administer competency-based, parallel pretests and posttests 
correlated with program goals and content to determine progress of 
participants toward mastery of instructional content 

Level HI evaluation criteria can be effectively met when program providers: 

♦ identify local indicators for measuring performance of specific behaviors on 
critical job tasks; and 

♦ collect ratings of performance, either by participants' supervisors or as 
employee self-ratings, on identified indicators pre- and post- 
program course cycles to measure the amount of transfer from training to 
job performance. 

Level IV evaluation criteria can be effr ctively met when employers and employees: 

♦ employers and employees examine areas in which critical job tasks are 
performed to determine the extent to which changes in performance after 
participation in workplace literacy training programs are saving or 
generating money for the organization. 

Most organizations use basic measurement of work output to meet product (or 
service) goals, emphasizing quality, quantity, variety, uniqueness of whatever is being 
produced or provided. Some common indicators that are measured include units produced, 
defects or failure rates, tasks completed, rework, scrap, backlogs, units sold, shortages, 
accidents, etc. There are also system goals which emphasize growth, profits, modes of 
functioning, and return on investment. Frequently used criteria are: productivity, 
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processing time, operating costs, efficiency, amounts of overtime or lost time, machine 
downtime, performance/cost ratio, etc. In addition, "hygiene" benefits like reduced 
turnover, absenteeism, strikes, etc., and reduction in accidents are also used to conduct 
cost benefit or cost effectiveness analyses. (Cascio, 1982; Kearsley, 1982). When data is 
already being collected for such measures, employers and employees may wish to examine 
it to identify relationships with positive changes in performance. Measures of individual 
productivity are easiest to correlate, but it is difficult to isolate training impact from other 
variables that may be producing a cost benefit Additionally, benefits can be rather diffuse 
and often take considerable time to be realized, which makes cost analysis somewhat 
inappropriate as a tool for evaluating training and development activities (Bramley, 1991). 

It also is quite difficult for external workplace literacy program providers to be 
permitted access to an organization's profit and loss or quality figures because of the 
privileged and secure status of such information. For this reason, most providers are well 
advised to encourage program evaluations through which they can establish Level m 
indicators that demonstrate the amount of transfer of learning to positive impact on 
performance, and leave cost benefit analysis that utilizes the information gathered to 
appropriate internal organizational staff. 

A means for costing changes in the workplace due to training can be found in the 
process of evaluating the value added to employees. The value-added approach assumes 
that there is a need for certain skills in areas relevant to the work performed, that there is 
motivation to do the job, and that there is an opportunity for the skills to be used to perform 
the job. Pre-training analysis must be conducted to estimate the position of individual 
targeted participants with respect to various levels of skill and motivation of typical 
employees who are doing that kind of work If employees are thought to have the 
motivation but lack some important skills, then the investment should be on employee skills 
training. Post program measures of skills and motivation are also collected and changes 
calculated Assuming that supervisors or managers can accurately rate their employees, 
that there is opportunity and encouragement to use the new skills, and that the skills 
required for successful job performance have been accurately identified and are being 
taught in training, value-added evaluation can be used to estimate the return on training 
investment. Unlike cost benefit analysis which summarizes all outcomes in monetary terms 
and omits those things that cannot be expressed in dollars, the value-added process 
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describes outcomes in their own terms and then concludes that these do or do not imply 
significant increase in skills or motivation (Bramley, 1991, pg.84). 

Whatever strategies are used to evaluate workplace literacy programs, it is critical 
that the data gathered and analyzed provides evidence that the program is or is not effective, 
based on the values of the organization. For many programs the time line for proving 
success is limited to o$b or two financial quarters. Companies are operating on lean 
budgets and are unable to justify training expenditures that do not demonstrate return on 
investment- minimally to Level in evaluation criteria, impact on performance* Without 
this kind of hard evidence that a workplace literacy program has a relationship to the bottom 
line of the organization, the program retains the status of a charitable effort Organizations 
sponsor workplace literacy programs for many reasons, ranging on a continuum from good 
works to improving public images to generating negotiable benefits to desire for job - 
performance improvements that address government regulations or impact on competition 
and survival. In lean times, programs that are perceived as charity are discontinued 

The true measure of program success is continuation and expansion through 
organizationally budgeted resources. For company-provided programs this entails constant 
searching for evidence each quarter that demonstrates the value of continuing the training 
program and looking for resources with which to expand it For external providers or 
recipients of demonstration project funds, this requires diligent efforts to develop 
meaningful workplace program indicators that demonstrate invaluable impact on the 
organization as well as individual participants so that the program will be institutionalized, 
budgeted, and incorporated into company training on an ongoing basis. 
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